Update results collection form to support pass/fail assertion verdicts #1003
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
howard-e
added a commit
that referenced
this pull request
Nov 30, 2023
* Add scripts to convert a test plan from v1 to v2 format * Add and revise support and commands JSON data support.json: * add strings to support screen reader settings and assertion tokens. * Add URL data for ARIA and HTML-AAM specifications. commands.json: Add all the command string representations necessary for the new V2 test format. * WIP: Building test index page * Adding additional TODOs to be covered; start to addressing review page * Fix references after rebase * Add additional data validations * Use MUST/SHOULD/MAY on review page * Update validations * Now accounting for modeless ATs * Additional validations * Accounting for assertionExceptions * Additional validations and accounting for assertionExceptions; review page format now also matches 977 * Keys handling in test-io-format.mjs * Update support.json * Remove 'v2-made' files * Change 'NA' for modeless to 'defaultMode' * Format .mjs files * Fix lines in keys.mjs * Remove create-example-tests2 * Revert keys.mjs quotations * assertionTokens replacement * show replaced assertion tokens in reviewed tests * Conditionally show modeInstructions * testing * Set mode for commandInfo * Better support for modeless in aria-at-test-io-format.mjs and aria-at-test-run.mjs * Better support for modeless in test-reviewer.mjs * Formatting * Update v2maker to create a new row for commands previously defined as 'F / Shift + F' for eg. * Avoid setupScript thrown error when looking for script file * Formatting test-reviewer.mjs * Handle unexpected cases with v2 maker related scripts * Handle thrown error csv properties to pass validations (for now) * Additional cleanup * Fix pattern name identifier on index pagae * Formatting * Handle all 'at'-commands.csv files without looking specifically for known ats * Additional error handling for missing at-commands.csv files * Update jsdocs * Formatting * Conditionally build v2 over v1 test format * Refactor name for clarity * Edge case handling for test with multiple settings * Optimize edge case handling * Misc * Fix showing mode instructions on review page * Update validations * Update error message * Revert changes in package.json * Fix issues with Delete command validation * Remove unused function * Ensure rows in *-commands.csv are maintained by the presentationNumber * Fix case where titles and instructions were still unexpectedly referencing reading and interactions modes in v2maker script. This work has been duplicated in #990 * Fix review page to stop using generated commandPresentationNumbers * Update aria-at-test-run.mjs to drop support for generated command sequence numbers * Better support of 0-level assertion exceptions on review page * Fix linter * Remove 'modes' from v2 test file names * Remove comment * Add mode to individual *.collected.(json|html) files * Fix '1 test's'' editorial error on review page * Fix for presenting .collected.html when needed * Add support for multiple settings with .collected * Don't show assertionPhrase column on review page for v1 build * Fix when commandListSettingsPreface gets appended * Change from {at}-{settings} .collected files to a single {at} .collected file which covers all settings * Fix combined .collected.(json|logic) * Update results collection form to support pass/fail assertion verdicts (#1003) * Initial rendering pass * Initial interaction with assertion checkbox * Cleanup and use assertionResponseQuestion * Remove unused required validation, Ensure input fires state change event * Use assertionResponseQuestion in v1 tests as well --------- Co-authored-by: Matt King <[email protected]> Co-authored-by: Stalgia Grigg <[email protected]> Co-authored-by: Stalgia Grigg <[email protected]>
howard-e
pushed a commit
that referenced
this pull request
Nov 30, 2023
#1003) * Initial rendering pass * Initial interaction with assertion checkbox * Cleanup and use assertionResponseQuestion * Remove unused required validation, Ensure input fires state change event * Use assertionResponseQuestion in v1 tests as well
howard-e
pushed a commit
that referenced
this pull request
Nov 30, 2023
#1003) * Initial rendering pass * Initial interaction with assertion checkbox * Cleanup and use assertionResponseQuestion * Remove unused required validation, Ensure input fires state change event * Use assertionResponseQuestion in v1 tests as well
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Preview Tests
see #969
blocked from merging by #997
This is an initial pass. These files are new to me so there is likely something I am missing (especially with output/serialization). I will leave this is as a draft for now and will have some side-channel conversations with others to ensure I am addressing everything.
Also, note that this is currently targeting
build-v2-test-formatto make viewing the diffs easier but this will targetmainonce that branch is merged.